use dmabuf option when the ibperf binaries have them configured#235
Open
ahskabir wants to merge 12 commits into
Open
use dmabuf option when the ibperf binaries have them configured#235ahskabir wants to merge 12 commits into
ahskabir wants to merge 12 commits into
Conversation
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
* Add CLAUDE.md with codebase guidance for Claude Code
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* Update CLAUDE.md with project structure and full lib/ listing
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* Add TorchTitan training tests, lib, config templates, and guide
- torchtitan_training_lib.py: TorchTitanTrainingJob class for single-node
and multi-node distributed training using torchrun + TOML config
- Test files: Llama 3.1 8B and 70B x single-node and multi-node
- Config templates: single-node and distributed JSON with model params
for mi300x and mi325; single-node nnodes hardcoded to 1
- TORCHTITAN_TEST_GUIDE.md: full reference for running and extending tests
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* Rewrite CLAUDE.md as Claude-focused guidance; minor torchtitan cleanup
- CLAUDE.md: restructure for editing-context (Conventions, workload-class
lifecycle, env vars, "Where to look when..."); drop generic defaults
already enforced by ruff; remove TORCHTITAN_TEST_GUIDE reference.
- torchtitan_training_lib.py: simplify tt_config to {module}_{model_size};
use self.master_address in single-node torchrun rdzv_endpoint;
collapse duplicate single-node docker exec loop.
- Delete TORCHTITAN_TEST_GUIDE.md.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
* Remove empty __init__.py from tests/training/torchtitan
Not required for pytest collection (megatron/ and jax/ siblings work
without it) nor for sdist packaging (MANIFEST.in uses recursive-include).
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
* torchtitan: generated TOML, config-driven root, robust launcher
- Add config-driven TOML generation: use_generated_config=True writes a
TOML from model_params to {scripts_dir}/run_config.toml; False falls
back to the canned TOML shipped with TorchTitan (path now includes
.toml extension).
- Parameterize torchtitan_root (default /workspace/torchtitan); 7 new
per-model fields (hf_assets_path, converters, dataset, lr,
warmup_steps, enable_async_tensor_parallel,
precompute_float8_dynamic_scale_for_fsdp) populate the generated TOML.
- Inline config_file_path into torchrun instead of exporting
\$CONFIG_FILE: the wrapper-script echo was expanding \$CONFIG_FILE
against the SSH session env at echo-time, silently overriding our
export with a stale /tmp path.
- Wrap download + sleep + torchrun chain in nohup sh -c '...' & disown
with stdio detached so it survives docker exec returning.
- Truncate training.log before download/sleep window so the poller
doesn't scan stale prior-run output during the launch gap; bump
pre-poll wait from 80s to 300s to clear download + sleep 200.
- Align with megatron reference: pattern-table-driven regex parsing
(TRAINING_RESULT_PATTERNS, TRAINING_PROGRESS_PATTERNS_TEMPLATE,
TRAINING_NAN_PATTERNS) + helpers, configurable hca_id_pattern with
re.escape per segment, fuller docstrings on lifecycle methods.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
* torchtitan configs: default use_generated_config to False
Switch the shipped templates to the canned-TOML path so customers get
the upstream TorchTitan preset out of the box. Customers who want full
JSON-driven config can flip this to "True".
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
* fmt: auto-format torchtitan_training_lib.py
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* remove unused TorchTitanLlamaTrainingJob alias
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* make fmt changes
* torchtitan: add qwen3_32b tests, sync HF download, log-format fixes
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* adding torchtitan deepseek_16b models test files and config files
* removed CLAUDE.md file
---------
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
Co-authored-by: Rajesh Thummala <rthummal@amd.com>
Co-authored-by: Rajesh Thummala <rthummal_amdeng>
…o resolve Tb errors Signed-off-by: Urvashi Tiwari <urtiwari.com>
Signed-off-by: Urvashi Tiwari <urtiwari.com>
Introduces an opt-in nginx reverse proxy layer (docker-compose.nginx.yml) that terminates TLS on port 443 and proxies to Grafana and Fleet Manager over the internal Docker network. Required for deployments on public IPs where Zscaler performs SSL inspection and blocks non-standard ports. - docker-compose.nginx.yml: compose override adding fleet-nginx container; overrides GF_SERVER_ROOT_URL via GRAFANA_ROOT_URL env var; mounts /etc/letsencrypt and /var/www/certbot for Let's Encrypt integration - server/config/nginx/nginx.conf: nginx config with ACME HTTP-01 challenge support on port 80, TLS termination on 443, WebSocket proxy headers for Grafana live, and /fleet/ path for Fleet Manager API - server/config/nginx/ssl/.gitkeep: placeholder keeps ssl/ dir in tree; certs are gitignored (*.pem, *.key already in .gitignore) - .env.example: documents three GRAFANA_ROOT_URL deployment tracks: private/VPN (no TLS), nip.io + Let's Encrypt (staging/test), AMD-issued cert (production) Validated on chi-metrics-exp-mon (137.220.56.231) with Let's Encrypt cert for 137.220.56.231.nip.io — clears Zscaler SSL inspection. Rollback (base stack untouched): docker compose -f docker-compose.yml -f docker-compose.nginx.yml rm -sf fleet-nginx Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces a Go-based SSH connection pool (go-collector) that maintains persistent connections per host, replacing the ephemeral Python parallel-SSH implementations (cvs_parallel_ssh_reliable, host_probe, jump_host_pssh). Adds ssh_manager.py and go_collector.py on the Python side to interface with the new daemon, and updates main.py and the Dockerfile/docker-compose accordingly. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace Python SSH layer with persistent Go SSH collector
cijohnson
reviewed
Jul 6, 2026
|
|
||
| try: | ||
| # detailed=True gives us exit_code per host | ||
| host_results = phdl.exec(cmd, detailed=True, print_console=False) |
Collaborator
There was a problem hiding this comment.
It assumes a homogeneous cluster — all nodes have the same binary at the same app_path. That's a safe assumption in the ibperf context (the binary path comes from a shared config and is the same for all nodes), and it's consistent with how the rest of the test already treats the cluster.
So we can just use shdl.exec instead of phdl.exec, shdl.exec will test the binary only in the head node(which is sufficent)
cijohnson
reviewed
Jul 6, 2026
|
|
||
| dmabuf_supported_any = False | ||
|
|
||
| for host, result in host_results.items(): |
Collaborator
There was a problem hiding this comment.
when shdl is used this whole for loop to set "dmabuf_supported_any " becomes unnecessary.
cijohnson
requested changes
Jul 6, 2026
* OMPI installation testcase
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
* OMPI build with suppor for optionally building UCX
- configuration for ompi installation is updated so that user can
provide information about UCX needs to be built
- ompi can be built with or without UCX after applying this patch
- RCCL test was run to test ompi built without ucx and it worked
- RCCL test was run to test ompi built with ucx and it worked
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
* ompi config file has now "changeme" in two places
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
---------
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
…e/perf_dmabuf Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
IBPERF test makes use of the binaries for which path is provided in the configuration file. It also has an option to install the
ibperftool and then to run the test. But during the run-time when it prepares theibperfinvocation command(s) it doesn't use the--use_rocm_dmabufoption and the run could be less performant.Technical Details
A function is added in the ibperf library. This function check to see whether the ibperf binary was configured with --use_rocm_dmabuf. If yes, it will use the flag for invocation, otherwise not.
Test Plan
All ibperf test was run using ibperf binaries configured with --use_rocm_dmabuf and test passed.
All ibperf test was run using ibperf binaries not configured with --use_rocm_dmabuf and test passed.
Test Result
In both the scenarios, all tests passed.
Submission Checklist